Large-scale computing systems, such as those associated with network-based production services, have become widely available in recent years. Examples of such systems include online merchants, internet service providers, online businesses such as photo processing services, corporate networks, cloud computing services, web-based hosting services, etc. These entities may maintain large numbers of computing devices (e.g., thousands of hosts) which are hosted in geographically separate locations and which are configured to process large quantities (e.g., millions) of client requests daily or even hourly. Complex systems may include many services that interact with one another in varied ways.
In many cases, these services have not been tested properly when the services are put into production to serve requests from real-world clients. As a result, services in production may fail to perform as designed under atypical conditions or even under typical conditions. For example, services often have service level agreements (SLAs) that the services are expected to respect. The SLAs may relate to latency, scalability, throughput, etc. It may be difficult to know whether any given SLA is being respected before a service is put into production. Because performance problems may arise only after a service is in production, it may be difficult and time-consuming to identify the source of the problems in the program code for the service.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning “having the potential to”), rather than the mandatory sense (i.e., meaning “must”). Similarly, the words “include,” “including,” and “includes” mean “including, but not limited to.”